A (Brief) Tour of AI Art

by Andrew Look

“Whatever you now find weird, ugly, uncomfortable, and nasty about a new medium will surely become its signature.

CD distortion, the jitteriness of digital video, the crap sound of 8-bit, all of these will be cherished and emulated as soon as they can be avoided. It’s the sound of failure.

So much modern art is the sound of things going out of control. Out of a medium, pushing to its limits and breaking apart.”

-- Brian Eno

image credit

Before we begin

You Are here because:

  • You are aware of AI but might not have heard about artistic uses
  • You're curious to learn how AI can augment the creative process

By the end of this tour

My goal is to help you understand:

  • How using AI art tools can augment your creativity
  • How some existing artists are incorporating AI tools and concepts into their work
  • Just enough tools, concepts, and aesthetic options to begin making your own AI art
  • Aesthetic controls at a deeper level, for one particular technique (deepdreams)
  • Where to learn more if you want to get involved!

About Me

I'm Andrew Look, ML engineer on the Knowledge team at Pinterest.

I work on building classifier algorithms to look at each pin and determine its Taste Graph Interests.

I started painting as a stress-relief hobby from startup life pre-Pinterest.

In [0]:
# asset ideas:
# - hacker companion painting
# - me asleep with book
# - pinterest yearbook

Disclaimer: not a real expert on computer vision, neuroscience or art

But I have spent a couple years learning about the space, so I'll do my best to explain things as I understand them so far.

Disclaimer: For those more familiar with AI

  • I plan on making some broad generalizations in the interests of brevity.
  • If you're interested in the technical details, I'll have resources for you afterwards!

Following along

If you don't have a laptop, this presentation is designed to still be interesting and useful.

Request: Audience volunteer to slack me an image I can use for demos: @look

If you have a laptop, you may follow along using this interactive notebook hosted on Google Colab: Start Here

Start here

All you need to do to follow along are a few steps:

  1. Save a copy of the Notebook in your own Google Drive account
  2. Change runtime to GPU
  3. Upload your photo(s) and display them
  4. Kick off the longer setup scripts and wait a few mins
  5. Test deepdream just to make sure everything works

Step 1 - Save a copy

If you want to save your notebook to you own Google Drive account.

In the top menu, choose File -> Save a copy in drive from the top menu when you're ready to save.

Step 2 - Change Runtime to "GPU"

This makes notebook run much, much faster.

In the top menu, Click Runtime and then select Change Runtime Type.

Then, click the Hardware Accelerator dropdown, select GPU and then click Save

In [9]:
ipyd.Image(filename='drive/projects/whirlwind/gpu_py2_colab.png', width=400)
Out[9]:

When you're done, running the following code cell should confirm that a GPU is available:

In [10]:
import tensorflow as tf
device_name = tf.test.gpu_device_name()
if device_name != '/device:GPU:0':
  raise SystemError('GPU device not found')
print('Found GPU at: {}'.format(device_name))


import sys
print(sys.version)
assert sys.version.startswith('2')
Found GPU at: /device:GPU:0
2.7.14 (default, Sep 23 2017, 22:06:14) 
[GCC 7.2.0]

Step 3 - Kick off the longer setup scripts and wait a few mins

There's a bunch of big blobs of code below - if you skip down to a cell below them and run the following, it will execute all cells before that one: <Cmd>-<fn>-F8 (Note: you may not need to press fn key).

Click here to jump ahead to the right cell.

In [1]:
# basic image mgmt functions


!pip install scikit-image lucid

from __future__ import print_function
import numpy as np
import PIL

from IPython import display as ipyd
from lucid.misc.io import reading, showing
from skimage import io
# http://scikit-image.org/docs/dev/api/skimage.transform.html
from skimage import transform
from IPython import display as ipyd

# from https://stackoverflow.com/questions/39382412/crop-center-portion-of-a-numpy-image
def crop_center(img, cropx, cropy):
    y, x, _ = img.shape
    startx = x // 2 - (cropx // 2)
    starty = y // 2 - (cropy // 2)    
    return img[starty:starty+cropy, startx:startx+cropx, :]
 
def square_shrink(img, dim):
  smaller_dim = min(img.shape[:2])
  cropped = crop_center(img, smaller_dim, smaller_dim)
  small_cropped = transform.resize(cropped, (dim, dim, img.shape[2]), mode='constant')
  return small_cropped

def load_img(fname, dim=None):
  img = io.imread(fname)
  if not dim:
    return transform.resize(img, (img.shape[0], img.shape[1], img.shape[2]), mode='constant')
  small_img = square_shrink(img, dim=dim)
  return small_img

def upload_images():
  from google.colab import files
  uploaded = files.upload()
  img_fnames = []
  for fn in uploaded.keys():
    print('User uploaded file "{name}" with length {length} bytes'.format(
        name=fn, length=len(uploaded[fn])))
    img_fnames.append(fn)
  return img_fnames
Collecting scikit-image
  Downloading https://files.pythonhosted.org/packages/26/5a/b95745a38ca32014f3604957a91f80d40a0aae9c6ea28a19048670be81b7/scikit_image-0.13.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (29.3MB)
    100% |████████████████████████████████| 29.3MB 30kB/s  eta 0:00:01
Requirement already satisfied: lucid in /Users/alook/code/lucid
Requirement already satisfied: scipy>=0.17.0 in /opt/conda/miniconda2/lib/python2.7/site-packages/scipy-1.1.0rc1-py2.7-macosx-10.5-x86_64.egg (from scikit-image)
Collecting PyWavelets>=0.4.0 (from scikit-image)
  Downloading https://files.pythonhosted.org/packages/da/2a/dd1b00eb1835e0f8d6f8b8f024fb77183cf1fe1d6910d95d48394cd80740/PyWavelets-0.5.2-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (4.8MB)
    100% |████████████████████████████████| 4.8MB 175kB/s eta 0:00:01
Collecting networkx>=1.8 (from scikit-image)
  Downloading https://files.pythonhosted.org/packages/11/42/f951cc6838a4dff6ce57211c4d7f8444809ccbe2134179950301e5c4c83c/networkx-2.1.zip (1.6MB)
    100% |████████████████████████████████| 1.6MB 584kB/s eta 0:00:01
Requirement already satisfied: six>=1.7.3 in /opt/conda/miniconda2/lib/python2.7/site-packages (from scikit-image)
Collecting matplotlib>=1.3.1 (from scikit-image)
  Downloading https://files.pythonhosted.org/packages/61/38/d70e8bf77d5cb27d5f3595edd0b3978825063feadd023786d2591e393e6e/matplotlib-2.2.2-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (13.7MB)
    100% |████████████████████████████████| 13.7MB 79kB/s  eta 0:00:01
Requirement already satisfied: pillow>=2.1.0 in /opt/conda/miniconda2/lib/python2.7/site-packages/Pillow-5.1.0-py2.7-macosx-10.5-x86_64.egg (from scikit-image)
Requirement already satisfied: future in /opt/conda/miniconda2/lib/python2.7/site-packages (from lucid)
Requirement already satisfied: decorator in /opt/conda/miniconda2/lib/python2.7/site-packages (from lucid)
Requirement already satisfied: tensorflow in /opt/conda/miniconda2/lib/python2.7/site-packages/tensorflow-1.8.0rc0-py2.7-macosx-10.5-x86_64.egg (from lucid)
Requirement already satisfied: scikit-learn in /opt/conda/miniconda2/lib/python2.7/site-packages/scikit_learn-0.19.1-py2.7-macosx-10.5-x86_64.egg (from lucid)
Requirement already satisfied: numpy in /opt/conda/miniconda2/lib/python2.7/site-packages/numpy-1.14.2-py2.7-macosx-10.5-x86_64.egg (from lucid)
Requirement already satisfied: pyopengl in /opt/conda/miniconda2/lib/python2.7/site-packages/PyOpenGL-3.1.1a1-py2.7.egg (from lucid)
Requirement already satisfied: ipython in /opt/conda/miniconda2/lib/python2.7/site-packages (from lucid)
Collecting cycler>=0.10 (from matplotlib>=1.3.1->scikit-image)
  Downloading https://files.pythonhosted.org/packages/f7/d2/e07d3ebb2bd7af696440ce7e754c59dd546ffe1bbe732c8ab68b9c834e61/cycler-0.10.0-py2.py3-none-any.whl
Requirement already satisfied: backports.functools-lru-cache in /opt/conda/miniconda2/lib/python2.7/site-packages (from matplotlib>=1.3.1->scikit-image)
Collecting subprocess32 (from matplotlib>=1.3.1->scikit-image)
  Downloading https://files.pythonhosted.org/packages/b8/2f/49e53b0d0e94611a2dc624a1ad24d41b6d94d0f1b0a078443407ea2214c2/subprocess32-3.2.7.tar.gz (54kB)
    100% |████████████████████████████████| 61kB 8.2MB/s eta 0:00:01
Collecting kiwisolver>=1.0.1 (from matplotlib>=1.3.1->scikit-image)
  Downloading https://files.pythonhosted.org/packages/79/d8/94633718f3f77dcb638687a77ba199325a1cb158d2d4b00c9dc17f2b5c72/kiwisolver-1.0.1-cp27-cp27m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (110kB)
    100% |████████████████████████████████| 112kB 7.9MB/s eta 0:00:01
Requirement already satisfied: python-dateutil>=2.1 in /opt/conda/miniconda2/lib/python2.7/site-packages (from matplotlib>=1.3.1->scikit-image)
Requirement already satisfied: pytz in /opt/conda/miniconda2/lib/python2.7/site-packages (from matplotlib>=1.3.1->scikit-image)
Collecting pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 (from matplotlib>=1.3.1->scikit-image)
  Downloading https://files.pythonhosted.org/packages/6a/8a/718fd7d3458f9fab8e67186b00abdd345b639976bc7fb3ae722e1b026a50/pyparsing-2.2.0-py2.py3-none-any.whl (56kB)
    100% |████████████████████████████████| 61kB 3.0MB/s eta 0:00:01
Requirement already satisfied: absl-py>=0.1.6 in /opt/conda/miniconda2/lib/python2.7/site-packages/absl_py-0.2.0-py2.7.egg (from tensorflow->lucid)
Requirement already satisfied: astor>=0.6.0 in /opt/conda/miniconda2/lib/python2.7/site-packages/astor-0.6.2-py2.7.egg (from tensorflow->lucid)
Requirement already satisfied: backports.weakref>=1.0rc1 in /opt/conda/miniconda2/lib/python2.7/site-packages/backports.weakref-1.0.post1-py2.7.egg (from tensorflow->lucid)
Requirement already satisfied: enum34>=1.1.6 in /opt/conda/miniconda2/lib/python2.7/site-packages (from tensorflow->lucid)
Requirement already satisfied: gast>=0.2.0 in /opt/conda/miniconda2/lib/python2.7/site-packages/gast-0.2.0-py2.7.egg (from tensorflow->lucid)
Requirement already satisfied: grpcio>=1.8.6 in /opt/conda/miniconda2/lib/python2.7/site-packages/grpcio-1.11.0-py2.7-macosx-10.5-x86_64.egg (from tensorflow->lucid)
Requirement already satisfied: mock>=2.0.0 in /opt/conda/miniconda2/lib/python2.7/site-packages/mock-2.0.0-py2.7.egg (from tensorflow->lucid)
Requirement already satisfied: protobuf>=3.4.0 in /opt/conda/miniconda2/lib/python2.7/site-packages/protobuf-3.5.2.post1-py2.7-macosx-10.5-x86_64.egg (from tensorflow->lucid)
Requirement already satisfied: tensorboard<1.8.0,>=1.7.0 in /opt/conda/miniconda2/lib/python2.7/site-packages/tensorboard-1.7.0-py2.7.egg (from tensorflow->lucid)
Requirement already satisfied: termcolor>=1.1.0 in /opt/conda/miniconda2/lib/python2.7/site-packages/termcolor-1.1.0-py2.7.egg (from tensorflow->lucid)
Requirement already satisfied: wheel in /opt/conda/miniconda2/lib/python2.7/site-packages (from tensorflow->lucid)
Requirement already satisfied: simplegeneric>0.8 in /opt/conda/miniconda2/lib/python2.7/site-packages (from ipython->lucid)
Requirement already satisfied: pickleshare in /opt/conda/miniconda2/lib/python2.7/site-packages (from ipython->lucid)
Requirement already satisfied: backports.shutil-get-terminal-size; python_version == "2.7" in /opt/conda/miniconda2/lib/python2.7/site-packages (from ipython->lucid)
Requirement already satisfied: pexpect; sys_platform != "win32" in /opt/conda/miniconda2/lib/python2.7/site-packages (from ipython->lucid)
Requirement already satisfied: pathlib2; python_version == "2.7" or python_version == "3.3" in /opt/conda/miniconda2/lib/python2.7/site-packages (from ipython->lucid)
Requirement already satisfied: traitlets>=4.2 in /opt/conda/miniconda2/lib/python2.7/site-packages (from ipython->lucid)
Requirement already satisfied: pygments in /opt/conda/miniconda2/lib/python2.7/site-packages (from ipython->lucid)
Requirement already satisfied: appnope; sys_platform == "darwin" in /opt/conda/miniconda2/lib/python2.7/site-packages (from ipython->lucid)
Requirement already satisfied: setuptools>=18.5 in /opt/conda/miniconda2/lib/python2.7/site-packages (from ipython->lucid)
Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.4 in /opt/conda/miniconda2/lib/python2.7/site-packages (from ipython->lucid)
Requirement already satisfied: futures>=2.2.0 in /opt/conda/miniconda2/lib/python2.7/site-packages (from grpcio>=1.8.6->tensorflow->lucid)
Requirement already satisfied: funcsigs>=1 in /opt/conda/miniconda2/lib/python2.7/site-packages (from mock>=2.0.0->tensorflow->lucid)
Requirement already satisfied: pbr>=0.11 in /opt/conda/miniconda2/lib/python2.7/site-packages/pbr-4.0.2-py2.7.egg (from mock>=2.0.0->tensorflow->lucid)
Collecting bleach==1.5.0 (from tensorboard<1.8.0,>=1.7.0->tensorflow->lucid)
  Downloading https://files.pythonhosted.org/packages/33/70/86c5fec937ea4964184d4d6c4f0b9551564f821e1c3575907639036d9b90/bleach-1.5.0-py2.py3-none-any.whl
Collecting html5lib==0.9999999 (from tensorboard<1.8.0,>=1.7.0->tensorflow->lucid)
  Downloading https://files.pythonhosted.org/packages/ae/ae/bcb60402c60932b32dfaf19bb53870b29eda2cd17551ba5639219fb5ebf9/html5lib-0.9999999.tar.gz (889kB)
    100% |████████████████████████████████| 890kB 928kB/s eta 0:00:01
Requirement already satisfied: markdown>=2.6.8 in /opt/conda/miniconda2/lib/python2.7/site-packages/Markdown-2.6.11-py2.7.egg (from tensorboard<1.8.0,>=1.7.0->tensorflow->lucid)
Requirement already satisfied: werkzeug>=0.11.10 in /opt/conda/miniconda2/lib/python2.7/site-packages (from tensorboard<1.8.0,>=1.7.0->tensorflow->lucid)
Requirement already satisfied: ptyprocess>=0.5 in /opt/conda/miniconda2/lib/python2.7/site-packages (from pexpect; sys_platform != "win32"->ipython->lucid)
Requirement already satisfied: scandir; python_version < "3.5" in /opt/conda/miniconda2/lib/python2.7/site-packages (from pathlib2; python_version == "2.7" or python_version == "3.3"->ipython->lucid)
Requirement already satisfied: ipython-genutils in /opt/conda/miniconda2/lib/python2.7/site-packages (from traitlets>=4.2->ipython->lucid)
Requirement already satisfied: wcwidth in /opt/conda/miniconda2/lib/python2.7/site-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython->lucid)
Building wheels for collected packages: networkx, subprocess32, html5lib
  Running setup.py bdist_wheel for networkx ... done
  Stored in directory: /Users/alook/Library/Caches/pip/wheels/44/c0/34/6f98693a554301bdb405f8d65d95bbcd3e50180cbfdd98a94e
  Running setup.py bdist_wheel for subprocess32 ... done
  Stored in directory: /Users/alook/Library/Caches/pip/wheels/31/d3/4d/8151bf3fba7d27d0c9d03b78d4ff16097a8b62b298dc937c8d
  Running setup.py bdist_wheel for html5lib ... done
  Stored in directory: /Users/alook/Library/Caches/pip/wheels/50/ae/f9/d2b189788efcf61d1ee0e36045476735c838898eef1cad6e29
Successfully built networkx subprocess32 html5lib
Installing collected packages: PyWavelets, networkx, cycler, subprocess32, kiwisolver, pyparsing, matplotlib, scikit-image, html5lib, bleach
  Found existing installation: html5lib 1.0.1
    Uninstalling html5lib-1.0.1:
      Successfully uninstalled html5lib-1.0.1
  Found existing installation: bleach 2.1.3
    Uninstalling bleach-2.1.3:
      Successfully uninstalled bleach-2.1.3
Successfully installed PyWavelets-0.5.2 bleach-1.5.0 cycler-0.10.0 html5lib-0.9999999 kiwisolver-1.0.1 matplotlib-2.2.2 networkx-2.1 pyparsing-2.2.0 scikit-image-0.13.1 subprocess32-3.2.7
You are using pip version 9.0.2, however version 10.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-9db9dc091a1c> in <module>()
      5 
      6 from __future__ import print_function
----> 7 import numpy as np
      8 import PIL
      9 

ImportError: No module named numpy
In [0]:
### Tensorflow deepdream model (inception5h)

# first, download a pretrained model to use
import os
if not os.path.exists('inception5h.zip'):
  !wget https://storage.googleapis.com/download.tensorflow.org/models/inception5h.zip && unzip inception5h.zip
In [13]:
### Tensorflow Deepdream Code
# - Source: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/tutorials/deepdream
# - Original notebook: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/examples/tutorials/deepdream/deepdream.ipynb
#
# Other references:
# - https://github.com/mtyka/deepdream_highres
#
# Source is inlcuded under Apache V2 License:
# - https://github.com/tensorflow/tensorflow/blob/master/LICENSE

# boilerplate code
from __future__ import print_function
import os
from io import BytesIO
import numpy as np
from functools import partial
import PIL.Image
from IPython.display import clear_output, Image, display, HTML

import tensorflow as tf

model_fn = 'tensorflow_inception_graph.pb'

# creating TensorFlow session and loading the model
graph = tf.Graph()
sess = tf.InteractiveSession(graph=graph)
with tf.gfile.FastGFile(model_fn, 'rb') as f:
    graph_def = tf.GraphDef()
    graph_def.ParseFromString(f.read())
t_input = tf.placeholder(np.float32, name='input') # define the input tensor
imagenet_mean = 117.0
t_preprocessed = tf.expand_dims(t_input-imagenet_mean, 0)
tf.import_graph_def(graph_def, {'input':t_preprocessed})

layers = [op.name for op in graph.get_operations() if op.type=='Conv2D' and 'import/' in op.name]
feature_nums = [int(graph.get_tensor_by_name(name+':0').get_shape()[-1]) for name in layers]

print('Number of layers', len(layers))
print('Total number of feature channels:', sum(feature_nums))


# Helper functions for TF Graph visualization

def strip_consts(graph_def, max_const_size=32):
    """Strip large constant values from graph_def."""
    strip_def = tf.GraphDef()
    for n0 in graph_def.node:
        n = strip_def.node.add() 
        n.MergeFrom(n0)
        if n.op == 'Const':
            tensor = n.attr['value'].tensor
            size = len(tensor.tensor_content)
            if size > max_const_size:
                s = "<stripped %d bytes>"%size
                tensor.tensor_content = str(s).encode('utf-8')
    return strip_def
  
def rename_nodes(graph_def, rename_func):
    res_def = tf.GraphDef()
    for n0 in graph_def.node:
        n = res_def.node.add() 
        n.MergeFrom(n0)
        n.name = rename_func(n.name)
        for i, s in enumerate(n.input):
            n.input[i] = rename_func(s) if s[0]!='^' else '^'+rename_func(s[1:])
    return res_def
  
def show_graph(graph_def, max_const_size=32):
    """Visualize TensorFlow graph."""
    if hasattr(graph_def, 'as_graph_def'):
        graph_def = graph_def.as_graph_def()
    strip_def = strip_consts(graph_def, max_const_size=max_const_size)
    code = """
        <script>
          function load() {{
            document.getElementById("{id}").pbtxt = {data};
          }}
        </script>
        <link rel="import" href="https://tensorboard.appspot.com/tf-graph-basic.build.html" onload=load()>
        <div style="height:600px">
          <tf-graph-basic id="{id}"></tf-graph-basic>
        </div>
    """.format(data=repr(str(strip_def)), id='graph'+str(np.random.rand()))
  
    iframe = """
        <iframe seamless style="width:800px;height:620px;border:0" srcdoc="{}"></iframe>
    """.format(code.replace('"', '&quot;'))
    display(HTML(iframe))

# Visualizing the network graph. Be sure expand the "mixed" nodes to see their 
# internal structure. We are going to visualize "Conv2D" nodes.
tmp_def = rename_nodes(graph_def, lambda s:"/".join(s.split('_',1)))
show_graph(tmp_def)
Number of layers 59
Total number of feature channels: 7548
In [14]:
## visualizing neuron activations


# start with a gray image with a little noise
img_noise = np.random.uniform(size=(224,224,3)) + 100.0

def showarray(a, fmt='jpeg'):
    a = np.uint8(np.clip(a, 0, 1)*255)
    f = BytesIO()
    PIL.Image.fromarray(a).save(f, fmt)
    display(Image(data=f.getvalue()))
    
def visstd(a, s=0.1):
    '''Normalize the image range for visualization'''
    return (a-a.mean())/max(a.std(), 1e-4)*s + 0.5

def T(layer):
    '''Helper for getting layer output tensor'''
    return graph.get_tensor_by_name("import/%s:0"%layer)

def render_naive(t_obj, img0=img_noise, iter_n=20, step=1.0):
    t_score = tf.reduce_mean(t_obj) # defining the optimization objective
    t_grad = tf.gradients(t_score, t_input)[0] # behold the power of automatic differentiation!
    
    img = img0.copy()
    for i in range(iter_n):
        g, score = sess.run([t_grad, t_score], {t_input:img})
        # normalizing the gradient, so the same step size should work 
        g /= g.std()+1e-8         # for different layers and networks
        img += g*step
        print(score, end = ' ')
    clear_output()
    showarray(visstd(img))
    
    
layer = 'mixed4d_3x3_bottleneck_pre_relu'
channel = 139 # picking some feature channel to visualize
render_naive(T(layer)[:,:,:,channel])
In [15]:
def tffunc(*argtypes):
    '''Helper that transforms TF-graph generating function into a regular one.
    See "resize" function below.
    '''
    placeholders = list(map(tf.placeholder, argtypes))
    def wrap(f):
        out = f(*placeholders)
        def wrapper(*args, **kw):
            return out.eval(dict(zip(placeholders, args)), session=kw.get('session'))
        return wrapper
    return wrap

# Helper function that uses TF to resize an image
def resize(img, size):
    img = tf.expand_dims(img, 0)
    return tf.image.resize_bilinear(img, size)[0,:,:,:]
resize = tffunc(np.float32, np.int32)(resize)


def calc_grad_tiled(img, t_grad, tile_size=512):
    '''Compute the value of tensor t_grad over the image in a tiled way.
    Random shifts are applied to the image to blur tile boundaries over 
    multiple iterations.'''
    sz = tile_size
    h, w = img.shape[:2]
    sx, sy = np.random.randint(sz, size=2)
    img_shift = np.roll(np.roll(img, sx, 1), sy, 0)
    grad = np.zeros_like(img)
    for y in range(0, max(h-sz//2, sz),sz):
        for x in range(0, max(w-sz//2, sz),sz):
            sub = img_shift[y:y+sz,x:x+sz]
            g = sess.run(t_grad, {t_input:sub})
            grad[y:y+sz,x:x+sz] = g
    return np.roll(np.roll(grad, -sx, 1), -sy, 0)
  
  
def render_multiscale(t_obj, img0=img_noise, iter_n=10, step=1.0, octave_n=3, octave_scale=1.4):
    t_score = tf.reduce_mean(t_obj) # defining the optimization objective
    t_grad = tf.gradients(t_score, t_input)[0] # behold the power of automatic differentiation!
    
    img = img0.copy()
    for octave in range(octave_n):
        if octave>0:
            hw = np.float32(img.shape[:2])*octave_scale
            img = resize(img, np.int32(hw))
        for i in range(iter_n):
            g = calc_grad_tiled(img, t_grad)
            # normalizing the gradient, so the same step size should work 
            g /= g.std()+1e-8         # for different layers and networks
            img += g*step
            print('.', end = ' ')
        clear_output()
        showarray(visstd(img))
        
        
render_multiscale(T(layer)[:,:,:,channel])
In [16]:
k = np.float32([1,4,6,4,1])
k = np.outer(k, k)
k5x5 = k[:,:,None,None]/k.sum()*np.eye(3, dtype=np.float32)

def lap_split(img):
    '''Split the image into lo and hi frequency components'''
    with tf.name_scope('split'):
        lo = tf.nn.conv2d(img, k5x5, [1,2,2,1], 'SAME')
        lo2 = tf.nn.conv2d_transpose(lo, k5x5*4, tf.shape(img), [1,2,2,1])
        hi = img-lo2
    return lo, hi

def lap_split_n(img, n):
    '''Build Laplacian pyramid with n splits'''
    levels = []
    for i in range(n):
        img, hi = lap_split(img)
        levels.append(hi)
    levels.append(img)
    return levels[::-1]

def lap_merge(levels):
    '''Merge Laplacian pyramid'''
    img = levels[0]
    for hi in levels[1:]:
        with tf.name_scope('merge'):
            img = tf.nn.conv2d_transpose(img, k5x5*4, tf.shape(hi), [1,2,2,1]) + hi
    return img

def normalize_std(img, eps=1e-10):
    '''Normalize image by making its standard deviation = 1.0'''
    with tf.name_scope('normalize'):
        std = tf.sqrt(tf.reduce_mean(tf.square(img)))
        return img/tf.maximum(std, eps)

def lap_normalize(img, scale_n=4):
    '''Perform the Laplacian pyramid normalization.'''
    img = tf.expand_dims(img,0)
    tlevels = lap_split_n(img, scale_n)
    tlevels = list(map(normalize_std, tlevels))
    out = lap_merge(tlevels)
    return out[0,:,:,:]

  
def render_lapnorm(t_obj, img0=img_noise, visfunc=visstd,
                   iter_n=10, step=1.0, octave_n=3, octave_scale=1.4, lap_n=4):
    t_score = tf.reduce_mean(t_obj) # defining the optimization objective
    t_grad = tf.gradients(t_score, t_input)[0] # behold the power of automatic differentiation!
    # build the laplacian normalization graph
    lap_norm_func = tffunc(np.float32)(partial(lap_normalize, scale_n=lap_n))

    img = img0.copy()
    for octave in range(octave_n):
        if octave>0:
            hw = np.float32(img.shape[:2])*octave_scale
            img = resize(img, np.int32(hw))
        for i in range(iter_n):
            g = calc_grad_tiled(img, t_grad)
            g = lap_norm_func(g)
            img += g*step
            print('.', end = ' ')
        clear_output()
        showarray(visfunc(img))

# Showing the lap_normalize graph with TensorBoard
lap_graph = tf.Graph()
with lap_graph.as_default():
    lap_in = tf.placeholder(np.float32, name='lap_in')
    lap_out = lap_normalize(lap_in)
show_graph(lap_graph)
In [17]:
render_lapnorm(T(layer)[:,:,:,channel])
In [18]:
render_lapnorm(T(layer)[:,:,:,65])
In [0]:
def render_deepdream(t_obj, img0=img_noise,
                     iter_n=10, step=1.5, octave_n=4, octave_scale=1.4):
    t_score = tf.reduce_mean(t_obj) # defining the optimization objective
    t_grad = tf.gradients(t_score, t_input)[0] # behold the power of automatic differentiation!

    # split the image into a number of octaves
    img = img0
    octaves = []
    for i in range(octave_n-1):
        hw = img.shape[:2]
        lo = resize(img, np.int32(np.float32(hw)/octave_scale))
        hi = img-resize(lo, hw)
        img = lo
        octaves.append(hi)
    
    # generate details octave by octave
    for octave in range(octave_n):
        if octave>0:
            hi = octaves[-octave]
            img = resize(img, hi.shape[:2])+hi
        for i in range(iter_n):
            g = calc_grad_tiled(img, t_grad)
            img += g*(step / (np.abs(g).mean()+1e-7))
            print('.',end = ' ')
        clear_output()
        showarray(img/255.0)        
In [20]:
# #### TF RUN CODE


# # Picking some internal layer. Note that we use outputs before applying the ReLU nonlinearity
# # to have non-zero gradients for features with negative initial activations.
layer = 'mixed4d_3x3_bottleneck_pre_relu'
channel = 139 # picking some feature channel to visualize
render_naive(T(layer)[:,:,:,channel])


render_multiscale(T(layer)[:,:,:,channel])
  




# # Lower layers produce features of lower complexity.
# render_lapnorm(T('mixed3b_1x1_pre_relu')[:,:,:,101])

# # optimizing a linear combination of features often gives a "mixture" pattern.
# render_lapnorm(T(layer)[:,:,:,65]+T(layer)[:,:,:,139], octave_n=4)
In [21]:
# Semantic Dictionary setup from:
# TODO(look): rem`ove my workaround once the py3 bud gets patched

# - https://github.com/tensorflow/lucid/blob/master/notebooks/building-blocks/SemanticDictionary.ipynb
# Used under Apache V2 License:
# ---------------------------------------------------------------------
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.



# TODO(look): remove my "workaround" once the py3 bud gets patched
# !pip uninstall lucid
# !rm -rf /opt/lucid /content/src/lucid && git clone https://github.com/andrewlook/lucid.git /opt/lucid && cd /opt/lucid/ && git checkout py3
# ! cd /opt/lucid && python setup.py --quiet install
# !ls /opt/lucid/lucid/scratch/web/svelte.py
# !cp /opt/lucid/lucid/scratch/web/svelte.py .
# import svelte as lucid_svelte
!pip install --quiet lucid
import lucid.scratch.web.svelte as lucid_svelte

!npm install -g svelte-cli@2.2.0

import numpy as np
import tensorflow as tf

import lucid.modelzoo.vision_models as models
import lucid.optvis.render as render
from lucid.misc.io import show, load
from lucid.misc.io.showing import _image_url
/tools/node/bin/svelte -> /tools/node/lib/node_modules/svelte-cli/bin.js
/tools/node/lib
└── svelte-cli@2.2.0 

In [22]:
%%html_define_svelte SemanticDict

<div class="figure">
    <div class="input_image">
        <div class="image" style="background-image: url({{image_url}}); z-index: -10;"></div>
        <svg class="pointer_container" viewBox="0 0 {{N[0]}} {{N[1]}}">

            {{#each xs as x}}
            {{#each ys as y}}
              <rect x={{x}} y={{y}} width=1 height=1
                class={{(x == pos[0] && y == pos[1])? "selected" : "unselected"}}
                on:mouseover="set({pos: [x,y]})"></rect>
            {{/each}}
            {{/each}}
        </svg>
    </div>
    <div class="dict" >
        {{#each present_acts as act, act_ind}}
        <div class="entry">
            <div class="sprite" style="background-image: url({{spritemap_url}}); width: {{sprite_size}}px; height: {{sprite_size}}px; background-position: -{{sprite_size*(act.n%sprite_n_wrap)}}px -{{sprite_size*Math.floor(act.n/sprite_n_wrap)}}px; --info: {{act.n}};"></div>
            <div class="value" style="height: {{sprite_size*act.v/1000.0}}px;"></div>
        </div>
        {{/each}}
    </div>
</div>


<style>
    .figure {
        padding: 10px;
        width: 1024px;
    }
    .input_image {
        display: inline-block;
        width: 224px;
        height: 224px;
    }
    .input_image .image, .input_image .pointer_constainer {
        position: absolute;
        width: 224px;
        height: 224px;
        border-radius: 8px;
    }
    .pointer_container rect {
      opacity: 0;
    }
    .pointer_container .selected {
      opacity: 1;
      fill: none;
      stroke: hsl(24, 100%, 50%);
      stroke-width: 0.1px;
    }
    
    .dict {
        height: 128px;
        display: inline-block;
        vertical-align: bottom;
        padding-bottom: 64px;
        margin-left: 64px;
    }
    .entry {
        margin-top: 9px;
        margin-right: 32px;
        display: inline-block;
    }
    .value {
        display: inline-block;
        width: 32px;
        border-radius: 8px;
        background: #777;
    }
    .sprite {
        display: inline-block;
        border-radius: 8px;
    }
    .dict-text {
        display: none;
        font-size: 24px;
        color: #AAA;
        margin-bottom: 20px;
    }
</style>

<script>
    
  function range(n){
    return Array(n).fill().map((_, i) => i);
  }
  
  export default {
    data () {
      return {
        spritemap_url: "",
        sprite_size: 64,
        sprite_n_wrap: 1e8,
        image_url: "",
        activations: [[[{n: 0, v: 1}]]],
        pos: [0,0]
      };
    },
    computed: {
      present_acts: (activations, pos) => activations[pos[1]][pos[0]],
      N: activations => [activations.length, activations[0].length],
      xs: (N) => range(N[0]),
      ys: (N) => range(N[1])
    },
    helpers: {range}
  };
</script>
Trying to build svelte component from html...
svelte compile --format iife /tmp/svelte_KV5Sz3/SemanticDict_1bf1f72.html > /tmp/svelte_KV5Sz3/SemanticDict_1bf1f72.js
svelte version 1.64.1
compiling ../tmp/svelte_KV5Sz3/SemanticDict_1bf1f72.html...

In [0]:
layer_spritemap_sizes = {
    'mixed3a' : 16,
    'mixed3b' : 21,
    'mixed4a' : 22,
    'mixed4b' : 22,
    'mixed4c' : 22,
    'mixed4d' : 22,
    'mixed4e' : 28,
    'mixed5a' : 28,
  }

def googlenet_spritemap(layer):
  assert layer in layer_spritemap_sizes
  size = layer_spritemap_sizes[layer]
  url = "https://storage.googleapis.com/lucid-static/building-blocks/googlenet_spritemaps/sprite_%s_channel_alpha.jpeg" % layer
  return size, url

googlenet = models.InceptionV1()
googlenet.load_graphdef()


def googlenet_semantic_dict_from_url(layer, img_url):
    img = load(img_url)
    googlenet_semantic_dict(layer, img)
    
# ## **User facing constructor**
# Now we'll create a convenient API for creating semantic dictionary
# visualizations. It will compute the network activations for an image,
# grab an appropriate spritemap, and render the interface.
def googlenet_semantic_dict(layer, img):    
    # Compute the activations
    with tf.Graph().as_default(), tf.Session():
        t_input = tf.placeholder(tf.float32, [224, 224, 3])
        T = render.import_model(googlenet, t_input, t_input)
        acts = T(layer).eval({t_input: img})[0]
    
    # Find the most interesting position for our initial view
    max_mag = acts.max(-1)
    max_x = np.argmax(max_mag.max(-1))
    max_y = np.argmax(max_mag[max_x])
    
    # Find appropriate spritemap
    spritemap_n, spritemap_url = googlenet_spritemap(layer)
    
    # Actually construct the semantic dictionary interface
    # using our *custom component*
    lucid_svelte.SemanticDict({
        "spritemap_url": spritemap_url,
        "sprite_size": 110,
        "sprite_n_wrap": spritemap_n,
        "image_url": _image_url(img),
        "activations": [[[{"n": n, "v": float(act_vec[n])} for n in np.argsort(-act_vec)[:4]] for act_vec in act_slice] for act_slice in acts],
        "pos" : [max_y, max_x]
    })
   
   
In [24]:
googlenet_semantic_dict_from_url("mixed4d", "https://storage.googleapis.com/lucid-static/building-blocks/examples/dog_cat.png")

Jump down here and run all setup cells at once

run the following, it will execute all cells before that one: <Cmd>-<fn>-F8 (Note: you may not need to press fn key).

In [0]:

In [25]:
# Showing the lap_normalize graph with TensorBoard
lap_graph = tf.Graph()
with lap_graph.as_default():
    lap_in = tf.placeholder(np.float32, name='lap_in')
    lap_out = lap_normalize(lap_in)
show_graph(lap_graph)
In [27]:
# First run <Cmd>-<fn>-F8, to run all cells before this one.

# While you're waiting, you can hit <Shift>-Enter in this cell,
# and it will generate some output to verify that everything got set
# up correctly.

### Test deepdream just to make sure everything works


# Picking some internal layer. Note that we use outputs before applying the ReLU nonlinearity
# to have non-zero gradients for features with negative initial activations.
layer = 'mixed4d_3x3_bottleneck_pre_relu'
channel = 139 # picking some feature channel to visualize



render_naive(T(layer)[:,:,:,channel])

render_multiscale(T(layer)[:,:,:,channel])

render_lapnorm(T(layer)[:,:,:,channel])

Step 4 - Upload your photo(s) and display them

In [0]:
# run this to display file upload form, and click "Choose Files"
# image_filenames = upload_images()
Upload widget is only available when the cell has been executed in the current browser session. Please rerun this cell to enable.
Saving j1.jpg to j1 (1).jpg
Saving j2.jpg to j2 (1).jpg
User uploaded file "j1.jpg" with length 64873 bytes
User uploaded file "j2.jpg" with length 124776 bytes
In [0]:
# since images can be pretty large (and slow down the notebook to render them),
# it's handy to have some lower-res versions to quickly experiment with.
small_imgs = [load_img(f, dim=224) for f in image_filenames]
# use a helper from lucid to display the images neatly:
showing.images(small_imgs, labels=image_filenames)

# also load the full images, in case you want a high-res deepdream
large_imgs = [load_img(f) for f in image_filenames]
showing.images(large_imgs, labels=image_filenames, w=400)

# for the rest of the notebook to go smoothly, we'll keep a pointer to the first
# images that we loaded:
small_img_0 = small_imgs[0]
large_img_0 = large_imgs[0]
j1.jpg
j2.jpg
j1.jpg
j2.jpg

Colab Hints

If you want to learn more about how to navigate Colab:

To get a brief overview of Colab's features:

In [0]:
# voiceover: back to slides <what is AI art?>

What is AI art?

Definition #1: Making creative use of outputs from an AI Tool

Definition #2: Leveraging how AI represents information to deliberately craft an Effect

Definition #3: Exploring the Concepts within AI and what they mean for us

Framework to discuss AI Art

Moving forwards, we'll use Tool, Effect, and Concept to interpret different examples of art.

But first...

What does AI actually mean?

AI is a term used in so many contexts, that it can be confusing for an audience.

Most mentions of AI can be bucketed into one of two broad categories:

  • General AI: The futuristic aspiration of computers that experience consciousness
  • Specialized AI: Systems using statistics to learn patterns from data in order to make predictions.

What kind of AI are we focusing on today?

The Specialized kind (also known as Machine Learning).

We'll be focused on artistic applications of techniques that have sprung out of real-world research.

Many artistic tools come from people striving to understand algorithms affect our lives.

Sorry to disappoint, but this tutorial won't cover robots that paint:

not this

Source: Deepmind blog pdf youtube

OK, so what is ML?

Machine Learning (usually) refers to a type of algorithm that:

  • learns patterns and relationships from training data
  • is built to perform a clearly-defined task.

A Simple Example

Imagine we want to predict housing prices.

We'd probably start with a spreadsheet of prices we know to be true, alongside features for each house that we expect to be related to the price.

housepricestable

Source: medium.com/@kabab

In [0]:
# new slide

If we plot price vs. size on this toy dataset, we can start to see a slope

housepricesscatter

Predicting price using the house size

If we were to draw a line through these points, the "slope" would be what you multiply the size by in order to get the price (plus a "bias" term, if the slope doesn't meet the Y-axis right at zero).

The objective of an ML algorithm is to find a line to draw through these points that minimizes the error in its predictions.

In [0]:
# new slide

You can visualize the error as the distance from each real data point's Y coordinate to the prediction: the line's value at the corresponding X coordinate.

errorbars

What "Learning" Means

The way ML algorithms "learn" is defined by their objective.

In this case, they can start with a proposed line to draw, calculate the error, and make a next guess about what the best line could be.

This process gets repeated, and if everything goes according to plan, the errors decrease.

iterativelearning

In [0]:
# consider basic neuron diagram...

What is art?

Out of respect to artists everywhere, I'm not going to try to define it.


What's relevant here is that art can be defined by whoever is creating it.

duchamp

"Fountain," Marcel Duchamp, 1917 Source: wikipedia

Back to AI Art

Definition #1: Making creative use of outputs from an AI Tool

Definition #2: Leveraging how AI represents information to deliberately craft an Effect

Definition #3: Exploring the Concepts within AI and what they means for us

These definitions actually mirror the progression of my own art practice.

I'll step through them in an attempt to share how my understanding of the space has evolved since I first started.

Definition #1: Making creative use of outputs from an AI Tool

My Work: a first use-case for an AI art tool

I started getting serious about learning ML sometime after starting to paint.

Around the same time, I took an art class about "Painting the Personal Narrative," and we needed to make a past / present series of still life paintings.

I started with a "present" painting, which I called "startup dinner" (a joking reference to my somewhat unhealthy eating habits left over from pre-Pinterest life).

startup_dinner

Img Setup

In [33]:
# !cp drive/projects/startup_breakfast/3-breakfast-normal-orig.jpg drive/projects/whirlwind
# !cp drive/projects/startup_breakfast/3-breakfast-normal.jpg drive/projects/whirlwind
# !cp drive/projects/startup_breakfast/4-breakfast-dream.jpg drive/projects/whirlwind

breakfast_normal_full = 'drive/projects/whirlwind/3-breakfast-normal-orig.jpg'
breakfast_normal_cropped = 'drive/projects/whirlwind/3-breakfast-normal.jpg'
breakfast_dream_cropped = 'drive/projects/whirlwind/4-breakfast-dream.jpg'

small_img_0 = load_img(breakfast_normal_cropped, dim=224)
large_img_0 = load_img(breakfast_normal_cropped)

image_filenames = [
    breakfast_normal_cropped,
    breakfast_dream_cropped,
]

# since images can be pretty large (and slow down the notebook to render them),
# it's handy to have some lower-res versions to quickly experiment with.
small_imgs = [load_img(f, dim=224) for f in image_filenames]
# use a helper from lucid to display the images neatly:
showing.images(small_imgs, labels=image_filenames)

# also load the full images, in case you want a high-res deepdream
large_imgs = [load_img(f) for f in image_filenames]
#showing.images(large_imgs, labels=image_filenames, w=400)

# for the rest of the notebook to go smoothly, we'll keep a pointer to the first
# images that we loaded:
small_img_0 = small_imgs[0]
large_img_0 = large_imgs[0]
drive/projects/whirlwind/3-breakfast-normal.jpg
drive/projects/whirlwind/4-breakfast-dream.jpg

Emulating Surrealism?

When we were asked to make a painting of the "past," the prompt was to make it "nightmarish."

I had another photo chosen already, which i jokingly titled "startup breakfast".

The only problem was, I didn't know how to make a dream effect.

In [35]:
ipyd.Image(filename=breakfast_normal_full, width=600)
Out[35]:

Enter Deepdreams

Thankfully, code for making deepdreams had just been announced by Google Research.

deepdream_examples

Running my first Deepdream

In [36]:
small_new_img = render_deepdream(tf.square(T('mixed4c')), small_img_0*255)
In [37]:
new_img = render_deepdream(tf.square(T('mixed4c')), large_img_0*255)

Painting it

In [0]:
# TODO convo explaining this to someone

How does it work?

In [0]:
# TODO neural nets
# TODO weight updates / backprop basics
# TODO explaining dream / featureviz

Example: pix2pix

Even without much context on how it works, Pix2Pix is fun to play with and yields wierd results.

We'll look at some more in-depth approaches later (time permitting), but for now this is just fun.

bread_cat

Based on: "Image-to-Image Translation with Conditional Adversarial Networks", Isola et al, 2017

Definition #2: Leveraging how AI represents information to deliberately craft an Effect

My Work: exploring high and low layers

How does it work?

In [0]:
# Visualizing the network graph. Be sure expand the "mixed" nodes to see their 
# internal structure. We are going to visualize "Conv2D" nodes.
tmp_def = rename_nodes(graph_def, lambda s:"/".join(s.split('_',1)))
show_graph(tmp_def)
In [38]:
mixed4d = 'mixed4d_3x3_bottleneck_pre_relu'
mixed4d_65 = render_lapnorm(T(mixed4d)[:,:,:,65])

# Lower layers produce features of lower complexity.
mixed3b_101 = render_lapnorm(T('mixed3b_1x1_pre_relu')[:,:,:,101])

# optimizing a linear combination of features often gives a "mixture" pattern.
combo__mixed4d_65__mixed3b_101 = render_lapnorm(T(mixed4d)[:,:,:,65]+T(mixed4d)[:,:,:,139], octave_n=4)
In [0]:
#### TF RUN CODE


render_lapnorm(T(layer)[:,:,:,65])

# Lower layers produce features of lower complexity.
render_lapnorm(T('mixed3b_1x1_pre_relu')[:,:,:,101])

# optimizing a linear combination of features often gives a "mixture" pattern.
render_lapnorm(T(layer)[:,:,:,65]+T(layer)[:,:,:,139], octave_n=4)

Example: "Face Space"

For example, recent generative algorithms have rapidly improved their ability both to learn latent spaces, and to generate images from any point in these latent spaces.

In fact, those spooky faces earlier came from a visualization of Eigenfaces, which sought to "learn" how to represent faces in a "latent space".

facespace

Source: Face Recognition using Eigenfaces, Turk et al, 1991

Example: Toposketch

Researchers have built tools to allow visual navigation of latent spaces in order to gain fine-grained control over the generated artifacts.

toposketch_man_loop

Example: Face Interpolation

When we interpolate between points in the space, generating images at each point along the way, it's striking that (almost) every point in between looks like a person.

One can't but help thinking that these algorithms are beginning to learn some fundamental truths about what we as humans all have in common.

celeb_1hr_man

Source: youtube

Paper: Progressive Growing of GANs for Improved Quality, Stability, and Variation, Karras et al, 2018

Definition #3: Exploring the Concepts within AI and what they means for us

Among other things, this could encompass exploring AI's relationships to:

  • our society
  • the individual
  • the nature of consciousness

My explorations of concepts

Example: SketchRNN

In [0]:
# TODO sketchrnn / DRAW / paint?

Example: Treachery of Imagenet